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ABSTRACT 

Monte Carlo studies explored the sampling 
characteristics of Cohen's d and three approximations to Cohen's d 
when used as average effect size measures in meta-analysis. Reviews 
of 10, 100 , and 500 studies (M) were simulated, with degrees of 
freedom (df) varied in seven steps from 8 to 58. In a two independent 
groups design, samples were obtained from populations whose mean 
differences represented a zero, small, medium, or large effect size. 
One thousand replications of studies within each of the, 84 
combinations of effect size, df, and M were conducted, and a mean and 
standard error were obtained for each combination of conditions. As 
expected, d was a positively biased estimator of effect size, 
overestimating by as much as 12 percent even with Hedges' correction 
factor. Surprisingly, the most unbiased estimator of effect size and 
the highest relative efficiency was obtained with the approximation 
to d computed from the obtained t and corrected according to Hedges. 
The approximations to d from the nonparametr ic statistic and the 
obtained conventional significance levels were not consistent 
estimators of effect size. These simulation results suggest that 
additional study of the behavior of effect size estimators should 
precede a more widespread application of meta-analysis. (Author) 
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Abstract 

Monte carlo studies explored the sampling 
characteristics of Cohen's d and three approximations 
to Cohen's d_ \dien used as average effect size measures 
in meta-analysis. Reviews of 10, 100, and 500 studies 
(M) were simulated, with degrees of freed an ( df ) varied 
in seven steps frcm 8 to 58. In a two independent 
groups design, samples were obtained frcm populations 
whose mean differences represented a zero, small, 
medium, or large effect size (^) . One thousand 
replications of studies within each of the 84 
combinations of ^, df, and M were conducted, and a mean 
and standard error were obtained for each combination 
of conditions. As expected, d was a positively biased 
estimator of ^, overestimating ^ by as much as 12% even 
with Hedges 1 correction factor. Surprisingly, the most 
unbiased estimator of ^ ana che highest relative 
efficiency was obtained with the approximation to d 
conputed from the obtained t and corrected according to 
Hedges. The approximations to d frcm the nonparamet ric 
statistic and the obtained conventional significance 
levels were not consistent estimators of The 
results of these simulations suggest that additional 
study of the behavior of effect size estimators should 
precede a more widespread application of meta-analysis. 
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Introduction 

Recent scientific endeavor is characterized by an 
exponential increase in the number of studies 
constituting a body of literature. Since the data 
items of such a literature are often disparately 
published and contain inconsistent findings, the 
extrication of relationships among explanatory 
variables is often difficult to achieve. The 
integrative surmary of the literature has consequently 
emerged as a more important facet of the scientific 
process. Recently, methods of secondary analysis, 
including cluster and probability approaches, and meta- 
analytic methods have been proposed to add a 
quantitative dimension to such literature reviews. 

The most common meta-analytic approach currently 
utilized seems to be the one proposed by Glass (1976) 
in which an average effect size is computed across 
studies in a variable domain. The most widely used 
estimate of effect size is the standardized mean 
difference, d, where d = (Xe - Xc)/Sc, described by 
Cohen in 1969. Hedges (1981), however, has formally 
demonstrated that d is a biased estimate of A , where A 
= (Ue - jjc) &c and represents the true effect size. 
Hedges proposed a correction factor, l-(3/(4 * df) -1), 
which, when multiplied by each d before averaginq, 
<xn rtvts t he bias. 
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Since the means and standard deviations necessary 
to compute d are often not available to the reviewer, 
approximations to d based on parametric and 
nonparametric statistics and reported significance 
levels have been suggested. Before the widespread 
adoption of these estimates of d occurs, however, their 
sampling characteristics should be understood. Thus 
the purpose to the Monte Carlo studies reported here 
was to empirically examine the behavior of d and the 
approximations of d, and their corrections according to 
Fledges, as estimators of ^. 

Method 

The studies simulated an experimental situation 
with two independent groups represented as two 
populations consisting cf normally distributed, 
ccciputer-generated randcm numbers having a variance of 
unity. The population designated as simulating the 
control group had a mean of 10.0; the experimental 
populations had means of 10.0, 10.2, 10.5, and 10. P, 
the first simulating a situation in which the null 
hypothesis is true and the other three corresponding to 
effect sizes that were considered to represent small, 
medium, and large effects, respectively. To examine 
Hie influence of the number of effect sizes entering 
Hie analysis, reviews were simulated with 10, 100, -,nd 
r >00 studies (M) . In order to determine the effects of 
varying degrees of freedom (df ) , each review was 

erJc .) 
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conducted on equal- n "studies" with sample sizes; of 5, 
10, 20, and 30 and unequal- n studies simulate by the 
pairs 10-15, 10-20, and 10-25. 

A FORTRAN program generated 1000 reviews for each 
unique combination of t , M and df , For each study 
vvithin each review, the following statistics were 
computed: d, a parametric approximation to d, d] , a 
nonparametric approximation d2, and an approximation 
based on significance level, d3 . '" The approximations 
were computed as t 1/n 1 + l/n 2, where t was the 
value of the obtained t., the value of t. at the 
significance level associated with the obtained u, and 
where t. was the value of t associated with the 
conventional level of significance exceeded by the 
obtained t. (.05 or .01) respectively, in addition, 
each of these effect size estimates was multiplied by 
Hedges' correction factor to produce d\ dl ' , d2 1 , and 
d3\ respectively. A mean corrected and uncorrected 
estimated effect size was computed for each review, 
and, subsequently means and standard erors were 
determined for the 1000 reviews within each of the 84 
cxmbi nations of conditions. 
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Results 

As expected frcm the worfc of Hedges, for all 
positive values of ^, d overestimated and the , 
magnitude of this bias was directly proportional to the 
value of 4 (see Tables) . Although the bias of the 
corrected d was less than that of d ., d 1 was still 
positively baised; in fact, d' overestimated ^ by as 
much as 13% for large values of ^ and small values of M 
and df . Surprisingly, the parametric approximation to 
d was a less biased estimator of ^ than d, dl 1 
deviating no more than 1.5% frcm ^ in any condition. 
Both d2 and d3 overestimated ^ for ^ = 0.0 and 0.2 and 
underestimated y\ for effect sizes of 0.5 and O.P. The 
magnitude of the bias of these latter two 
approximations was very large for the very small and 
very large effect sizes, and Hedges' correction factor 
exacerbated this bias for both measures. Although d 
and its parametric approximation showed consistency 
across df and M, neither d2 nor d3 appeared to be a 
consistent estimator of All estimators showed very 
poor relative efficiency for small values of df and M, 
and in all cases, the corrected estimate had only a 
sligl ?$a^.^ ller standard .error than the uncorrected 
one. The parametric approximation, furthermore, was <i 
more efficient estimator than d. 
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Conclusions 

Several conclusions seem warranted by these 
findings, (a) In the long run, the use of a mean £, 
corrected or otherwise, in integrative literature 
reviews will overestimate the magnitude of the 

> 

relationship between explanatory variables* (b) The 
corrected parametric approximation to d is the least 
biased of . the heretofore suggested estimators of the 
true effect size, (c) The nonparametric approximation 
to (3 and the approximation based on the achieved 
conventional significance level are not consistent 
estimators of the true effect size and thus have less 
utility than the other effect size estimators, (d) Due 

£ 

to the poor efficiency of all the estimators, the 
meta-analyst should exercise extrane caution vshen 
reviewing a small nunber of studies characterized by 
small df. In general, the results of these simulations 
suggest that additional study of the behavior of effect 
size estimates should precede a more widespread 
appl ice*tion of meta-analysis. 
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Deviations 


of dl ' from ^ = 


.0 




DF 


M=10 


M=100 


M=500 


8 


0.0057 


0.0034 


0.0017 


18 


0.0023 


0.0020 


-0.0005 


23 


-0.0010 


-0.0008 


-0.0001 


28 


0.0015 


-0.0014 


-0.0010 


33 


0.0001 


0.001.1 


0.0005 


38 


-0.0019 


0.0001 


-0.0002 


58 


0.0029 


0.0012 


0.0001 
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Deviations of dl ' iron ^ = .2 



DF ' 


M=10 


M=100 


M=500 


8 


0.0042 


0.0032 


0.0018 


18 


0.0018 


0.0020 


-0.0004 


23 


-0.0012 


-0.0008 


0.0000 


28 


0.0015 


-0.0014 


-0.0010 


33 


0.0000 


0.0011 


O.OOOf 


38 


-0.0019 


0.0001 


-0.0002 


58 


0.0029 


0.0012 


0.0001 
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Deviations of dl ' from ^ = . 5 



DF 

« 


M=10 


M=100 

> 


M=500 


8 


•• 

0.0021 


0.0028 


0.0019 


IP 


0.003 0 


0.0018 


-0.0003 


23 


-0.0016 


-0.0010 


0.0001 


28 


-0.0014 


-0.0014 


-0.0005 


33 


-0.0001 


0.0012 


O.OOC* 


38 


-0.0019 


0.0001 


-0.0001 


58 


0.0029 


0.0012 


0.0002 
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Deviations of dl ' from \ - .8 



DF 


M=10 


M=100 


M=500 


8 


0.0000 


0.0024 


0.0020 


18 


0.0003 


0.0017 


-0.0003 


23 


0.0019 : 


-0.0011 


0.0002 


28 


0.0014 


-0.0014 


-0.0008 


33 


-0.0003 


0.0012 


0.0007 


38 


-0.0019 


0.0000 


0.0000 


58 


0.0029 


0.0012 


0.0002 



f 
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Deviations of d2' from ^ = .0 



DF 


M=10 


M=100 


M=500 




8 


0 . 3601 


0.3668 


0.3688 




IP 


0.2910 


0.2887 


0.2877 




23 


0*2673 


0.2697 


0.2704 




28 


0.2551 


0.2543 

r 


0.2525 




33 


0 . 2495 


0.2476 


0.2471 




38 


0.2151 


0.2130 


0.2128 




58 


0.1752 


0.176 


0.1763 
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Deviations of 62' from \ = .5 



DF M=10 M=100 M=500 

8 -0.0548 -0.0517 -0.0509 

18 -0.1149 -0.1147 -0.1155 

23 -0.1309 -0.1298 -0.1302 

28 -0.1415 -0.1419 -0.1420 

33 -0.1446 -0.1457 -0.1458 

38 -0.1665 -0.1642 -0.1650 

58 -0.1796 -0.1809 -0.1812 



ERIC 



9 

ERIC 



Page 13 

Deviations of 62 ' from A = .P 



DF M=10 M=100 M=500 



8 -0.2438 -0.2422 -0.2415 

18 -0.2762 -0.2791 -0.2805 

23 -0.2950 -0.2937 -0.2932 

28 -0.3009 -0.2997 -0.2932 

33 -0.3039 -0.3008 -0,3009 

38 -0.3104 -0.3086 -0.3087 

58 -0.3169 -0.3172 -0.317P 
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Deviations of d3 1 from ^ = .0 



DF 


M=10 


M=100 


M=500 


8 


0.3067 


0.8124 


0.8168 


18 


0.2475 


0.5850 


0.5880 


23 


0.2102 


0.5330 


0.5381 


28 


0.2041 


0.5084 


0.5120 


33 


0.1990 


0.4924 


0.4948- 


38 


0.1763 


0.4155 


0.4186 


58 


0.1372 


0.3406 


0.3423 
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Deviations of d3 1 from £ = .2 



DF 


M=10 


M=100 


M=500 


8 


0.1542 


0.6172 


0.6194 


18 


0.0977 


0.3915 


0.3924 


23 


0.0925 


0.3434 


0.3431 




0 OQP1 






33 


0.0918 


0.3001 


0.3000 


38 


0.0675 


0.2239 


0.2237 


58 


0.0554 


0.1484 


0.1484 
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Deviations cf d3 ' from _^ = .5 



DF 


M=10 


M=100 


M=500 


a 


0.0002 


0.3286 


0.3285 


18 


0.1099 


0.1030 


0.1029 


23 


0.0055 


0.0547 


0.0543 


28 


0.0285 


0.0285 


0 . 0281 


33 


-0.0104 


0.0124 


0.0122 


38 


-0.0697 


-0.0632 


-0.0632 


58 


-0.1369 


-0.1374 


-0.1375 
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Deviations of d3 ' from ^ = .8 



DP 


M=10 


M=100 


M=500 


8 


-0.2378 


0.0375 


0.3382 


18 


-0.1129 


-0.1835 


-0.1837 


23 


-0.0677 


-0.1835 


-0.2316 


28 


-0.0445 


-0.2569 , 


-0.2569 


33 


-0.027] 


-0.2726 


-0.2726 


38 


-0.0467 


-0.3467 


-0.3467 


58 


-0.1203 


-0.4203 


-0.4204 



